Lecture Notes on the EM Algorithm

This is a short tutorial on the EM algorithm, including modern proofs of monotonicity, and several examples focusing on the use of EM to handle heavy-tailed models (Laplace, Student) and on finite mixture estimation. 1 The Algorithm Consider a general scenario in which we have observed data x, and a set of unknown parameters θ. Let us also assume some prior p(θ) for the parameters, which could well be a flat prior. The a posteriori probability function p(θ|x) is proportional to p(x|θ)p(θ). Now, suppose that finding the MAP estimate of θ would be easier if we had access to some other data y, that is, it would be easy to maximize p(x,y|θ)p(θ), where p(x,y|θ) is related to p(x|θ) via marginalization p(x|θ) = ∫ p(y,x|θ) dy. (1) The expectation-maximization (EM) algorithm is an iterative procedure which can be shown to converge to a (local) maximum of the marginal a posteriori probability function p(θ|x) = p(x|θ) p(θ), without the need to explicitly manipulate the marginal likelihood p(x|θ). EM was first proposed in [9], and since then it has attracted a great deal of interest and stimulated a considerable amount of research. For example, the EM algorithm has been often used